Skip to content

feat: preserve finding relationships in scan history - #574

Open
mldangelo-oai wants to merge 4 commits into
mainfrom
mdangelo/codex/scan-comparison-storage
Open

feat: preserve finding relationships in scan history#574
mldangelo-oai wants to merge 4 commits into
mainfrom
mdangelo/codex/scan-comparison-storage

Conversation

@mldangelo-oai

@mldangelo-oai mldangelo-oai commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

Preserve confirmed, uncertain, and related finding relationships in the existing SQLite workbench. The existing matcher and CLI remain compatible; #575 builds the replacement matcher on this storage layer.

Changes

  • Validate complete comparison results before replacing saved data, and keep related findings separate from confirmed duplicates.
  • Reuse stable identities and confirmed links while preserving uncertainty, coverage rules, and saved comparison payloads.
  • Add two indexes and batch relationship lookups through an append-only migration.
  • Stream comparison JSON through UTF-8 stdin. Older custom plugins retain the original confirmed/uncertain format.
  • Keep legacy scan records without target IDs scoped to their repository.
  • Simplify alias bookkeeping to one parent map and preserve main's workbench stdin API and Python-resolution fixes.
  • Exercise large comparison payloads and run Python probes through stdin instead of long command-line arguments.

There are no public CLI changes or new model calls in this PR.

Testing

  • Focused history, publication-store, runtime, comparison, and CLI compatibility tests: 187 passed, 13 platform skips.
  • Full SDK suite, Bun 1.3.14 with seed 12345: 1,510 passed, 29 platform/environment skips, zero failures.
  • Type checking, formatting, Ruff on all four changed Python modules, and git diff --check passed.
  • Build, pack, and installed-package inspection passed, including all 241 archive entries, 109 bundled plugin files, CLI execution, and nested-worker startup. The packed history module matches the reviewed source.
  • The legacy-repository regression failed before the fix and passed afterward.
  • A 10,000-match payload passed through stdin. A temporary argv mutant failed with E2BIG; the restored implementation passed.
  • The simplified alias implementation matched the original across 203 deterministic graph cases, including a 10,000-link chain. Disposable verification probes are not included in the PR.

Risk and rollout

The append-only migration adds indexes without rewriting saved comparisons or completed scan artifacts. Index creation can hold SQLite's normal write lock on a large history. Older custom plugins need an upgrade to save related links or comparisons too large for command-line arguments.

Merge this PR before #575. This update merges main without rewriting branch history.

Public disclosure review

  • No customer, partner, prospect, or user identities, data, or identifying details are included.
  • No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
  • I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

@github-actions github-actions Bot added the enhancement New feature or request label Aug 20, 2026
@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the current head, cd4b7a67c7c729fac3e4ccc1c33c963abe974374. This is the scan-history storage and compatibility portion of #567. The existing matcher remains in place on this branch.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: cd4b7a67c7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

The description now includes the final QA results and merge order. Please review the current head, cd4b7a67c7c729fac3e4ccc1c33c963abe974374. The code has not changed since the earlier review.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: cd4b7a67c7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Aug 20, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-08-21T06:22:33.114290Z 67d380e Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the current head, 02a2e338f9f515a7a7d76d38a17e238bcbd32be0. The PR description includes the latest cleanup and QA results.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 02a2e338f9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the current head, aa2ed4119c0334b896cef05c426ec1b3a5fea86f. This additive update merges current main, including #573. The PR description has the fresh compatibility and full-suite QA results. Please check migration compatibility, confirmed-link reuse, uncertainty, and coverage handling.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: aa2ed4119c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ianw-oai
ianw-oai previously approved these changes Aug 21, 2026
Keep legacy repositories separate when preparing matching inputs, simplify finding alias bookkeeping, and retain the shared workbench stdin API. Use platform-aware Python probes and exercise comparison payloads larger than command-line limits.
@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review exact head 67d380ec17b191bbf31258d0fd6ef4873676de47 against main. This additive update preserves main's stdin and Python-resolution fixes, scopes legacy records without target IDs by repository, simplifies finding-alias bookkeeping, and exercises large comparison payloads. Please check relationship grouping, coverage and uncertainty, migration compatibility, and the transport fallback for older plugins.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: 67d380ec17

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants